Apparatus and method for controlling queuing of data at a node on a network

ABSTRACT

An approach to queuing data at a network switching node includes assigning a queue value such as a priority value or a weight to each of a plurality of queues which temporarily store data packets to be forwarded toward a link on the network. A bandwidth value related to a maximum amount of data to be transferred on the link is defined. A data limit value is assigned to each of the queues to limit the amount of data that can be forwarded from each queue. The data limit value is derived from the bandwidth value for the link. By placing bandwidth-related limits on the queues, low-priority queues are prevented from being dominated by high-priority queues, that is, low-priority queues are not prevented from forwarding their data by the priority structure. Also, the bandwidth-related limits allow for control of traffic flow to prevent overloading lower-capacity links on the network.

FIELD OF THE INVENTION

The invention relates generally to the field of digital communicationsand more particularly to systems and methods for transferring packets ofdata in a switching node used in a digital data network and forcontrolling queuing of data at the switching node.

BACKGROUND OF THE INVENTION

Digital networks have been developed to facilitate the transfer ofinformation including data and programs among digital computer systemsand numerous other types of devices. A variety of types of networks havebeen developed and implemented using diverse information transfermethodologies. In modern networks, information is transferred through amesh of switching nodes which are interconnected by communication linksin a variety of patterns. The mesh interconnection pattern can allow fora number of paths to be available through the network from each computersystem or other device to another computer system or other device.

Information transferred from a source device to a destination device isgenerally transferred in the form of fixed or variable-length datapackets, each of which is in general received by a switching node over acommunication link and transmitted over another communication link tofacilitate transfer of the packet to the destination device or toanother switching node along a path to the destination device. Eachpacket typically includes address information including a source addressthat identifies the device that generated the packet and a destinationaddress that identifies the particular device or devices which are toreceive the packet. Transfer of data packets on the network typicallytakes place in accordance with a packet transfer protocol. In commonnetworks such as the Internet, a protocol used is the common InternetProtocol (IP).

Typically, a switching node includes one or more input ports, each ofwhich is coupled to a communication link on the network to receive datapackets, and one or more output ports, each of which is coupled to acommunication link on the network to transmit packets. Each nodetypically also includes a switching fabric that couples data packetsfrom the input ports to the output ports for transmission.

A typical switching node also includes one or more packet queues at theinput and/or output side of the node for temporarily storing packetsthat are to be transferred onto the network. Input-queued switches havethe queues at the input ports, and output-queued switches have thequeues at the input ports. In either case, each link to which an outputport is coupled can be associated with multiple queues.

The configuration of the queuing structure is determined by the queuingapproach implemented in the node. For example, under a priority queuingapproach, the queues associated with a particular output can be definedaccording to a predetermined packet transfer priority scheme. One queuemay be assigned to store only the highest priority packets. Anotherqueue may store only the next highest level priority packets, and so on.When a packet is to be transferred onto the associated network link, ifa packet is stored in the highest priority queue, then that packet istransferred. If no packet is present in that queue, then the next queueis checked, and so on. This approach ensures that the highest prioritypackets will always be transferred first.

In a weighted queuing approach, such as Weighted Fair Queuing (WFQ),each queue is given a predetermined weight value. Packets are thentransferred out of the queues according to their weights. For example,in a system with four queues for an output port, a weighting scheme canbe used in which the four queues are assigned weight values of 8, 4, 2,and 2. The queue with the weight value of 8 is permitted to forward 8/16or one half of the total traffic on the associated link. The queue withthe weight of 4 can forward one fourth of the traffic on the link. Insuch a system, if one of the queues has no traffic, then its permittedtraffic is allocated among the remaining queues. For example, if the oneof the queues with a weight of 2 has no traffic, then the queue with theweight of 8 can now transfer 8/14 of the total allowed link traffic.

In systems which implement one or both of these packet queuingapproaches, it is possible that packets with relatively low prioritywill never be transferred, or will be substantially delayed, if thetraffic of higher-priority packets is high. It is also possible that theamount of traffic permitted to be transferred from the queues on thelink may exceed the capabilities of other lower-capacity links on thenetwork, such as those carrying edge router data traffic.

SUMMARY OF THE INVENTION

A queuing approach according to the present invention provides alimitation on the amount of data traffic that can be taken from a singledata queue in a switching node such that lower-priority packets will beallowed to be transferred out of the node and the overall traffic rateout of each of the queues in the node can be limited to a predeterminedlevel. Limitations placed on the traffic out of the node can be based ona predetermined maximum data rate, i.e., bandwidth, of a link on thenetwork to which the node is coupled.

The present invention is directed to an apparatus and method forcontrolling data traffic at a node on a network. The node includes atleast one output coupled to at least one link on the network. Abandwidth value is defined for the at least one link and is related to amaximum amount of data to be transferred on the link. A plurality ofqueues are provided for storing data to be transferred on the link. Eachqueue is assigned a queue value which is related to a preference fortransferring data from the queue to the link. In addition, each queue isalso assigned a data limit value used to limit the amount of data thatcan be transferred out of the queue to the link. The data limit value isderived from the bandwidth value defined for the link. In one particularembodiment, the data limit value assigned to each queue is related to apercentage of the bandwidth value.

The link whose bandwidth value is used to apply limits to the queues canbe the link that is directly connected to the output of the switchingnode. Alternatively, it can be any selected link on the network towardwhich packets are to be transferred and whose traffic carrying capacityis to be considered. For example, the link may carry traffic for arouter near an edge of the network. As such, its traffic carryingcapacity may be relatively low when compared to that of a core link. Inthis case, applying limits related to the bandwidth of the link to thequeues in accordance with the invention can prevent this relativelydistant link from being overloaded.

In one embodiment, the invention implements an improved priority queuingapproach. In this embodiment, the queue values assigned to the queuesare priority values, each of which defines a packet transfer priorityfor its associated queue. Hence, in this embodiment, the inventionapplies a data traffic limit based on a predetermined link bandwidthvalue to each queue in addition to packet transfer priority valuesassigned to each queue.

In another embodiment, the invention implements an improved weightedqueuing approach. In this embodiment, each of the queue values defines aweight for its assigned queue, and the weights are used in determiningwhich packets should be transferred out of the queues. Hence, in thisembodiment, the invention applies a data traffic limit based on apredetermined link bandwidth value to each queue in addition to packettransfer weight values assigned to each queue.

The present invention provides advantages over prior approaches tocontrolling queuing of data such as priority queuing and weightedqueuing. For example, the bandwidth-related limitation on each queueprevents higher-priority queues in a priority-based queuing approachfrom dominating lower-priority queues to the extent that thelower-priority queues are unable to transfer data efficiently. Byapplying a limit related to the bandwidth of the link toward whichpackets are being transferred, high-priority queues are prevented fromtransferring all of their packets without limit, regardless of thecondition of lower-priority queues. As a result, the lower-priorityqueues are allowed to transfer their data more efficiently. Also, thebandwidth-related limitation allows for control of traffic such thatlower-capacity links are not overloaded.

The invention is applicable in various types of networks. For example,the invention can be implemented in a switching node such as the onedescribed in copending U.S. patent application Ser. No. 09/108,771,filed on Jul. 2, 1998, entitled “System and Method for Switching Packetsin a Network,” by Schwartz, et al., and assigned to the same assignee asthe present application. The contents of that application areincorporated herein in their entirety by reference.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other objects, features, and advantages of theinvention will be apparent from the following more particulardescription of preferred embodiments of the invention, as illustrated inthe accompanying drawings in which like reference characters refer tothe same parts throughout the different views. The drawings are notnecessarily to scale, emphasis instead being placed upon illustratingthe principles of the invention.

FIG. 1 contains a schematic diagram of a network including a pluralityof switching nodes in accordance with the present invention.

FIG. 2 contains a schematic block diagram of a switching node inaccordance with the present invention.

FIG. 3 contains a schematic block diagram of one embodiment of queuingcircuitry in accordance with the invention.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS OF THE INVENTION

FIG. 1 schematically depicts a computer network 10 including a pluralityof switching nodes 11(1) through 11(N), generally identified byreference numeral 11, for transferring signals representing data among anumber of devices, which in FIG. 1 are represented by packetsource/destination devices 12(1) through 12(M), generally identified byreference numeral 12, in a wide area network (“WAN”). The packetsource/destination devices 12 can include a particular device such as acomputer system or other device that stores, generates, processes orotherwise uses digital data. It can also be a local area network of suchdevices. Each packet source/destination device 12 is connected over acommunication link, generally identified by reference numeral 13, to aswitching node 111 to facilitate transmission of data thereto or thereception of data therefrom.

The switching nodes 11 are interconnected by communication links, alsogenerally identified by reference numeral 13, to facilitate the transferof information among the switching nodes. The communication links 13 mayutilize any convenient information transmission medium, including, forexample, wires for carrying electrical signals, optical fiber links forcarrying optical signals, and so forth. Each communication link 13 ispreferably bidirectional, allowing the switching nodes 11 to transmitand receive signals among each other and with customer premisesequipment 12 connected thereto over the same link. Depending on theparticular type of medium selected for respective communication links13, multiple media may be provided for transferring signals in oppositedirections thereby to provide the bidirectional link.

Data are transferred in the network 10 in the form of packets.Generally, a packet includes a header portion and a data portion. Theheader portion includes information that assists in routing the packetthrough the network, with the specific information depending on theparticular packet routing protocol that is used in routing packetsthrough the network. In connection with network 10, any of a number ofwell-known packet routing protocols may be used; in one embodiment, thewell-known Internet protocol (“IP”) is used. In any case, the headertypically includes address information including a source addressidentifying the particular source device 12(m _(S)) that generated thepacket and a destination address identifying the particular destinationaddress 12(m _(D)) that is to receive the packet. A packet may be ofvariable length and the header typically also includes lengthinformation to identify the length of the packet. The header alsotypically includes other information, including, for example, protocolidentifier information that identifies the particular protocol thatdefines the structure of the packet. The data portion contains the datapayload of the packet. The packet may also include, as part of the dataportion or otherwise, error detection information that can be used todetermine whether an error occurred in transferring the packet.

A source device 12(m _(D)), after generating a packet for transfer to adestination device 12(m _(D)), provides the packet to the switching node11(n) to which it is connected. The switching node 11(n) uses thedestination address in the packet to attempt to identify a route overwhich it is to transfer the packet to forward the packet to either thedestination device 12(m _(D)), if the switching node 11(n) is connectedto the destination device 12(m _(D)), or to another switching node11(n)(n′≈n) along a path to the destination device 12(m _(D)). The routeassociates a destination address with one of the communication links 13connected to the switching node. If the switching node can identify aroute for the received packet, it will forward the packet over thecommunication link identified by the route. Each switching node 11(n),11(n″), . . . that receives the packet will perform a similar operation.If all of the switching nodes have respective routes for the destinationaddress, the packet will eventually arrive at the destination device12(m _(D)).

FIG. 2 is a schematic block diagram of one embodiment of a switchingnode 11 in accordance with the present invention. The node 11 in generalincludes one or more input port modules 20 and one or more output portmodules 21. The input port module 20 and output port module 21 areconnected to processing circuitry and switching fabric 24 which controlsforwarding of data from the input port module 20 to the output portmodule 21. In general, each input port module 20 includes one or moreinput ports 22(1) through 22(N), which can be connected to communicationlinks 13. Likewise, each output port module 21 includes one or moreoutput ports 23(1) through 23(N) which can in general be connected tomultiple communication links 13. Data received on each of the links 13are forwarded from the associated input port 22 of the input port module20, across the processing circuitry and switching fabric 24, to theappropriate output port 23 of the output port module 21 and out onto thenetwork on the appropriate link 13.

As shown in FIG. 2, the input port module 20 and the output port module21 can also include queuing circuitry 27 and 25, respectively, inaccordance with the present invention. In the description of theinvention contained herein, the invention will be described in terms ofan output port module 21 with the queuing circuitry 25. It will beunderstood that the queuing circuitry 25 need not be located in theoutput port module 21 and also that the invention is applicable to anyqueuing circuitry used to store packets being transferred on a network.

The queuing circuitry 25 is used to queue data arriving at the outputport module 21 from the processing circuitry and switching fabric 24.The queuing circuitry 25 temporarily stores data packets before they aretransferred to network communication links 13 via output ports 23. Datapackets are transferred to the queuing circuitry 25 on lines 15. Datapackets out of the queuing circuitry 25 are forwarded to the outputports 23 on lines 17.

FIG. 3 is a schematic block diagram of one embodiment of the queuingcircuitry 25 shown in the output port module 21 in FIG. 2. As shown inFIG. 3, each output line 17 from the queuing circuitry is associatedwith one or more queues 31, 32, 33, 34. In the particular exemplaryembodiment shown in FIG. 3 and described herein, four queues 31, 32, 33,34 are associated with each output line 17. It will be understood thatany number of queues can be associated with an output. Also, each outputneed not be associated with the same number or type of queue.

The function of each queue is determined by the queuing approachimplemented for the particular associated output. One possible queuingapproach to which the invention is applicable is priority queuing, inwhich each queue is assigned a particular priority value. For example,using priority queuing, the queues 31–34 can be assigned prioritynumbers 1–4, respectively, with 1 representing the highest priorityqueue and 4 representing the lowest priority queue. Each packet isassociated with a priority value. For example, in one or more commonnetwork protocols, each packet carries what is commonly referred to as a“discard eligibility” (DE) value, which identifies the packet's priorityfor discard, i.e., for not being transferred out of the node onto thenetwork. Typically, packets with high DE values are considered morelikely to be discarded, and therefore have lower priority, than packetswith low DE values. Therefore, in one form of priority queuing, packetswith higher DE values are temporarily stored in the lower priorityqueues, e.g., the queue assigned priority 4, and packets with low DEvalues are stored in the higher priority queues, e.g., the queueassigned priority 1.

Under a conventional priority queuing approach, when a packet is to beremoved from a queue for transfer onto the network, the highest priorityqueue, i.e., the queue with priority value 1, is checked first for data.One or more packets in that queue are transferred onto the network. Ifno packets are stored in that queue, then the next priority queue, i.e.,the queue with value 2, is checked, and any packets stored in that queueare transferred. This continues until no more packets can be presentlytransferred onto the network or the queue with priority 4 is reached.

It will be appreciated that, under this approach, it can occur that thepriority 4 queue is not reached, and, as a result, none of the lowerpriority packets are forwarded. This can occur in particular duringhigh-traffic bursts of higher-priority traffic. While it is desirablethat all of the higher-priority traffic be forwarded, it is notdesirable to have very little or none of the lower-priority trafficforwarded through the node.

To address this problem, in accordance with the invention, all of thequeues can be limited in the amount of traffic they can send. In oneembodiment, the limitation is based on the bandwidth of a link on thenetwork. The link can be any link such as the link to which theassociated output is connected or a link on the network having lowercapacity than the local links. In one embodiment, the bandwidth-basedlimitations are expressed as a percentage of the bandwidth valueassociated with the pertinent link. For example, in one particularconfiguration, the queues can be assigned percentages of the bandwidthand priority values according to Table 1 below.

TABLE 1 Queue Reference Numeral 31 32 33 34 Priority  1  2  3  4Bandwidth Percentage 10 40 80 80As shown in Table 1, queue 31 has the highest priority, and packets aretaken from queue 31 first for transfer onto the network. However, thatpriority is limited to the extent that the packet traffic taken fromqueue 31 cannot exceed 10 percent of the bandwidth of the link.Therefore, packets will be taken from queue 31 up to the 10 percentlimit. When that limit is reached, packets will be taken from queue 32next. If there are no packets in queue 32, or if the 40 percent limit ofqueue 32 is reached, then packets will be taken from queue 33. If thereare no packets left in queue 33, or if the 80 percent limit of queue 33is reached, then packets will be taken from queue 34.

Thus, in accordance with this improved priority queuing approach of theinvention, packets from all of the queues will in general be permittedto be forwarded onto the network. In particular, lower-priority trafficwill still be forwarded, even in the presence of high-traffic bursts ofhigher-priority traffic. The bandwidth-constrained queuing approach ofthe invention provides for this while still maintaining thepredetermined priority scheme of the system.

It should be noted that this bandwidth-based limitation of the inventionis preferably applied to a queue only when there is traffic suitable fortransmission from some other queue associated with the same output link.That is, the bandwidth limit is preferably not applied if no othertraffic from any other queue is waiting for transmission. This approachavoids the potential for under-utilization of the link when no trafficis waiting.

In another embodiment, the invention is applicable to a packet fairqueuing approach such as weighted fair queuing (WFQ) to provide animproved weighted queuing approach with bandwidth-related constraintssimilar to those described above. Referring again to FIG. 3, under aweighted queuing approach, each of the queues is assigned a weight valueused to indicate its relative importance with respect to the otherqueues. For example, the queues 31, 32, 33, 34 can be assigned theweights 8, 4, 2, 2, respectively. Since the total of the weights is 16,queue 31 can forward 8/16, or one half, of the total traffic on theassociated output. Queue 32 can forward 4/16, or one eighth, and queues33 and 34 each can forward one eighth of the total traffic.

In such a system, when one of the queues has no traffic, that queue'straffic capacity is allocated among the other queues. For example, ifqueue 33 has no traffic, then the total of the weights becomes 14, andthe amount of traffic permitted to be forwarded by each of the otherqueues increases. For example, under these conditions, queue 31 can nowforward 8/14 of the total data traffic capacity. Thus, the amount oftraffic carried by the queues can increase substantially, especially ifone of the queues experiences a burst in traffic. This can be anespecially serious problem if a highly-weighted queue, such as queue 31in FIG. 3, experiences a significant traffic burst, since links on thenetwork can be overloaded.

This can be the situation where the queues are in a switching node atthe core of the network and are weighted to provide traffic on linkswith very high capacity. The traffic being forwarded may be intended tobe sent to an edge router via some other downstream link that does nothave the same traffic handling capacity. For example, the local corelink may have a 2.4 Gbits/sec capacity or bandwidth. However, the edgelink to which a portion of the traffic is to be forwarded may only havea 0.6 Gbits/sec bandwidth. Traffic out of the core switching nodeintended for that lower-capacity link should be limited to preventoverloading the distant link.

In accordance with the present invention, the traffic is limited at thecore switching node by placing additional traffic limits on the relevantqueues. These additional limits are based on a percentage of the linkbandwidth value as described above to ensure that links are notoverloaded. In one embodiment, the bandwidth value used to derive thelimits is the bandwidth value of the local core link. In the exampledescribed above, the pertinent queue may be limited to 25 percent of thetotal local link capacity such that the remote lower-capacity link isnot overloaded. In an alternative embodiment, the limit is based on apercentage of some other link such as the remote lower-capacity link.

Table 2 illustrates a general example of the bandwidth-constrainedlimits applied to the weighted queues in the improved weighted queuingapproach of the invention.

TABLE 2 Queue Reference Numeral 31 32 33 34 Weight  8  4  2  2 TrafficAllocation 8/16 4/16 2/16 2/16 Bandwidth Percentage 80 50 50 25As shown in Table 2, each queue is permitted to forward trafficaccording to its weighted traffic allocation, up to the bandwidthconstraint imposed in accordance with the invention. That is, in theillustrated example, queue 31 can forward one half of the total outputtraffic, as long as it does not exceed 80 percent of the bandwidth ofthe relevant link. Queue 32 can forward one fourth of the traffic up to50 percent of the bandwidth, queue 33 can forward one eighth of thetraffic up to 50 percent of the bandwidth, and queue 34 can forward oneeighth of the traffic up to 25 percent of the bandwidth.

The additional bandwidth constraints apply even if the amount of trafficallocated to the queues increases because one or more of the queues hasno traffic. This ensures that the link whose bandwidth is being used toderive the limits will not be overloaded, regardless of how traffic isallocated among the queues.

It should be noted that the link bandwidth value used to derive thelimits can in general differ from queue to queue. That is, multiplequeues for the same output can have bandwidth constraints derived fromthe bandwidths of different links, because, in general, queues can beassigned to store packets intended for transfer to particular links.

Also, as in the previously described embodiment, the limitations arepreferably only applied to a queue when there is traffic waiting inanother queue for transmission on the relevant link.

It should also be noted that the particular values, such as priorityvalues, weights and bandwidth percentage constraints set forth hereinare intended to serve as illustrative examples only and not as limitingparticular configurations of the invention described herein. It will beunderstood that many combinations of values for any of these variablesare possible, based on the particular packet transfer and queuingenvironment in which the invention is implemented.

While this invention has been particularly shown and described withreferences to preferred embodiments thereof, it will be understood bythose skilled in the art that various changes in form and details may bemade therein without departing from the spirit and scope of theinvention as defined by the appended claims.

1. A method of controlling data traffic at a node on a network, saidnode having at least one output coupled to at least one link on thenetwork, said method comprising: defining a bandwidth value for the atleast one link related to a maximum amount of data to be transferred onthe link; providing a plurality of queues for storing data to betransferred on the at least one link; assigning a queue value to each ofthe plurality of queues, each of said queue values being related to apreference for transferring data from its assigned queue onto the link;and assigning a data limit value to each of the plurality of queues, anamount of data being transferred onto the link from each queue beinglimited by the data limit value assigned to the queue, and each of saiddata limit values being derived from the bandwidth value for the atleast one link, wherein if one of the plurality of queues has no datatraffic, said queue's data traffic capacity is allocated among the otherqueues; characterized in that the queue value is selectively assigned toa specific queue based on preferences for transferring the data from itsassigned queue onto the link, and further characterized in that afterthe initial queue value assignment, further changes to the queue valuefor each queue may be selectively changed based on other preferences fortransfer of the data onto the link.
 2. The method of claim 1, whereinthe queue values are priority values, each priority value defining apriority for transferring data stored in the associated queue onto thelink.
 3. The method of claim 2, wherein each of the data limit valuesassigned to the queues is related to a percentage of the bandwidthvalue.
 4. The method of claim 1, wherein each of the queue valuesdefines a weight for its assigned queue, the plurality of queuestransferring data onto the link according to the weights of theplurality of queues.
 5. The method of claim 4, wherein each of the datalimit values assigned to the queues is related to a percentage of thebandwidth value.
 6. The method of claim 1, wherein each of the datalimit values assigned to the queues is related to a percentage of thebandwidth value.
 7. An apparatus for controlling data traffic at a nodeon a network, comprising: an output interface coupled to at least onelink on the network for transferring data toward the at least one link,the at least one link being associated with a bandwidth value related toa maximum amount of data to be transferred on the link; a plurality ofqueues for storing data to be transferred via the output interface; andat least one processor for (i) assigning a queue value to each of theplurality of queues, each of said queue values being related to apreference for transferring data from its assigned queue onto the link,and (ii) assigning a data limit value to each of the plurality of queuessuch that an amount of data being transferred onto the link from eachqueue is limited by the data limit value assigned to the queue, each ofsaid data limit values being derived from the bandwidth value for the atleast one link (iii) reallocating the data limit value from at least oneof said plurality of queues among the other queues when the at least onequeue has no data traffic; characterized in that the queue value isselectively assigned to a specific queue data based on preferences fortransferring the data from its assigned queue onto the link, and furthercharacterized in that after the initial queue value assignment, furtherchanges to the queue value for each queue may be selectively changedbased on other preferences for transfer of the data onto the link. 8.The apparatus of claim 7, wherein the queue values assigned by the atleast one processor are priority values, each priority value defining apriority for transferring data stored in the associated queue onto thelink.
 9. The apparatus of claim 8, wherein each of the data limit valuesassigned by the at least one processor is related to a percentage of thebandwidth value.
 10. The apparatus of claim 7, wherein: each of thequeue values assigned by the at least one processor defines a weight forits assigned queue; and the plurality of queues transfers data onto thelink according to the weights of the plurality of queues.
 11. Theapparatus of claim 10, wherein each of the data limit values assigned bythe at least one processor is related to a percentage of the bandwidthvalue.
 12. The apparatus of claim 7, wherein each of the data limitvalues assigned by the at least one processor is related to a percentageof the bandwidth value.